1 Introduction¶
The objective of this project is to do an extensive analysis of the Pulsepoint Emergency Data as well as apply some clustering and dimensionality reduction techniques.
The result from the analysis might be beneficial to a varieties of business stakeholders.
For example –
- For real estate business agencies, they may make decisions based on emergency occurrences and the frequency of which place is risky for housing and which are not and take precautions properly.
- For local authorities to avoid planting oil/gas filling stations fire prone locations.
- It will also be helpful for the local government to estimate a proper budget to take preventive measures for local emergencies and other natural phenomena.
Background¶
PulsePoint is a 911-connected mobile app that allows users to view and receive alerts on calls being responded to by fire departments and emergency medical services. The app's main feature, and where its name comes from, is that it sends alerts to users at the same time that dispatchers are sending the call to emergency crews. The goal is to increase the possibility that a victim in cardiac arrest will receive cardiopulmonary resuscitation (CPR) quickly. The app uses the current location of a user and will alert them if someone in their vicinity is in need of CPR. The app, which interfaces with the local government public safety answering point, will send notifications to users only if the victim is in a public place and only to users that are in the immediate vicinity of the emergency. - Wikipedia
Pulsepoint logs of the incidents can be used to identify the local pattern of emergencies which is helpful for local businesses as well as emergency agencies to stay alert and take precautions which, in the long term ensure the social well-being of the people.
Data Collection¶
The dataset was collected via web scraping using python. The logs were collected from 2021-05-02 to present. I uploaded the dataset on kaggle
PulsePoint Respond Mobile APP UI (visual inspection of the data) :
NB: This project also serves as my assignments for the course below -
View this project on GitHub : ahmedshahriar/PulsePoint-Data-Analytics¶
Kaggle Notebook : ahmedshahriarsakib/pulsepoint-emergency-analytics¶
Kaggle Dataset : ahmedshahriarsakib/pulsepoint-respond-local-threats-and-emergency¶
2 Libraries & Configuration¶
I used positionstack for geocoding data as a backup option for Nominatim¶
You can create an account on positionstack API (25,000 free requests/month)¶
In [ ]:
%%capture
!pip install geopandas # geo-plotting
!pip install pdpipe # data pipeline
!pip install yellowbrick # for elbow method
In [ ]:
import re
import json
import requests
import urllib
import pandas as pd
import numpy as np
import pdpipe as pdp
# from tqdm import tqdm
from tqdm.auto import tqdm # for notebooks
# Create new `pandas` methods which use `tqdm` progress
# (can use tqdm_gui, optional kwargs, etc.)
tqdm.pandas() # https://stackoverflow.com/a/34365537/11105356
from datetime import timedelta, datetime
# data visualization
import folium
import plotly.graph_objects as go
import plotly.express as px
import geopandas
import seaborn as sns
import matplotlib.pyplot as plt
from plotly.subplots import make_subplots
from wordcloud import WordCloud
from folium.plugins import MarkerCluster, HeatMap
from geopy.geocoders import Nominatim # reverse geocoding
# data processing and algorithm
from yellowbrick.cluster import KElbowVisualizer
from sklearn.cluster import (KMeans, DBSCAN, OPTICS,
AgglomerativeClustering,
MiniBatchKMeans)
from sklearn.preprocessing import MinMaxScaler, LabelEncoder
from sklearn.decomposition import PCA
from IPython.display import Image, HTML, Markdown
# from IPython.html import widgets
%matplotlib inline
sns.set(style='whitegrid', palette='muted', font_scale=1.2)
plt.rcParams['figure.figsize'] = 12, 8
# utility function to print markdown string
def printmd(string):
display(Markdown(string))
pd.set_option('display.max_colwidth', None)
SEED = 42
# set the size of the geo bubble
def set_size(value):
'''
Takes the numeric value of a parameter to visualize on a map (Plotly Geo-Scatter plot)
Returns a number to indicate the size of a bubble for a country which numeric attribute value
was supplied as an input
'''
result = np.log(1+value)
if result < 0:
result = 0.1
return result
# API Key
API_KEY_POSITIONSTACK = "YOUR_API_KEY_HERE"
3 Explore Dataset¶
In [ ]:
parse_dates=['date_of_incident']
pulse_point_df = pd.read_csv("/content/PulsePoint_local_threats_emergencies.csv",
parse_dates=parse_dates,
skipinitialspace=True)
# to parse datetime column later
# pulse_point_df.date_of_incident = pd.to_datetime(pulse_point_df.date_of_incident)
In [ ]:
printmd(f"Dataset has **{pulse_point_df.shape[0]}** rows and **{pulse_point_df.shape[1]}** columns")
Strip Object Columns
It will remove noise like extra whitespaces.
Example - there are some state values in the data such as - " CA" and "CA" which can be identified as separate entities. So this operation will remove that issue.
In [ ]:
pulse_point_df = pulse_point_df.applymap(lambda x: x.strip() if isinstance(x, str) else x)
In [ ]:
pulse_point_df.sort_values(by='date_of_incident')
Out[ ]:
Data was collected from 2021-05-02 to present
3.1 Metadata Summary¶
In [ ]:
pulse_point_df.columns
Out[ ]:
| Columns | Description | Data Type |
|---|---|---|
| id | Contains record id | numeric, int |
| type | Incident type (recent or active) | object |
| title | Title of the incident (e.g., Medical Emergency, Fire) | object |
| agency | Agency name (e.g., fire departments, emergency medical services) | object |
| location | Location where the incident took place | object |
| timestamp_time | Time when the incident record was logged | object |
| date_of_incident | Date when the incident record was logged | datetime |
| description | Emergency code description (e.g., E53 - refers to Fire Engine Truck ) | object |
| duration | Duration of the incident | object |
| Incident_logo | Logo of the incident | object |
| agency_logo | Logo of the agency | object |
In [ ]:
pulse_point_df.info()
Data Types¶
In [ ]:
pulse_point_df.dtypes.value_counts()
Out[ ]:
Object Data¶
In [ ]:
pulse_point_df.describe(include='object').T
Out[ ]:
Web Image Data¶
In [ ]:
def path_to_image_html(path):
'''
This function essentially convert the image url to
'<img src="'+ path + '"/>' format. And one can put any
formatting adjustments to control the height, aspect ratio, size etc.
within as in the below example.
'''
return '<img src="'+ path + '" style=max-height:124px;"/>' # option : '" width="60"
pulse_point_df_short = pulse_point_df.head(10)
HTML(pulse_point_df_short.to_html(escape=False , formatters=dict(incident_logo=path_to_image_html, agency_logo=path_to_image_html)))
Out[ ]:
| id | type | title | agency | location | timestamp_time | date_of_incident | description | duration | incident_logo | agency_logo | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 174293 | recent | Medical Emergency | OCFA (Div 2) | EXECUTIVE PK, IRVINE, CA | 3:05 AM | 2021-09-16 | CAR8799 E28 | 19 m |  |
|
| 1 | 174294 | recent | Medical Emergency | OCFA (Div 5) | LOS ADORNOS, ALISO VIEJO, CA | 3:05 AM | 2021-09-16 | CAR7199 E222 | 35 m | |
|
| 2 | 174295 | recent | Medical Emergency | Oceanside FD | DIXON RD, OCEANSIDE, CA | 3:00 AM | 2021-09-16 | E216 RA216 | 1 h 50 m | |
|
| 3 | 174296 | recent | Medical Emergency | OCFA (Div 7) | TULARE ST, BUENA PARK, CA | 3:00 AM | 2021-09-16 | CAR9361 E61 | 28 m | |
|
| 4 | 174297 | recent | Medical Emergency | OCFA (Div 6) | S NEWHOPE ST, SANTA ANA, CA | 2:59 AM | 2021-09-16 | CAR9578 E78 | 38 m | |
|
| 5 | 174298 | recent | Medical Emergency | Olathe Fire | W 124TH TER, OLATHE, KS | 2:58 AM | 2021-09-16 | M1151 S52 | 1 h 2 m | |
|
| 6 | 174299 | recent | Medical Emergency | OCFA (Div 4) | NEWPORT AVE, TUSTIN, CA | 2:57 AM | 2021-09-16 | CAR6526 E121 | 31 m | |
|
| 7 | 174300 | recent | Medical Emergency | OCFA (Div 5) | GREENLAWN CIR, LAGUNA NIGUEL, CA | 2:55 AM | 2021-09-16 | CAR7599 E39 | 22 m | |
|
| 8 | 174301 | recent | Medical Emergency | OCFA (Div 2) | GIOTTO, IRVINE, CA | 2:53 AM | 2021-09-16 | CAR6547 E6 | 21 m | |
|
| 9 | 174302 | recent | Smoke Detector | OCFA (Div 6) | 1801 S GREENVILLE ST, SANTA ANA, CA (VALLEY HIGH SCHOOL) | 2:52 AM | 2021-09-16 | E77 | 2 m |  |
3.2 Missing Values¶
In [ ]:
def missing_value_describe(data):
# check missing values in the data
total = data.isna().sum().sort_values(ascending=False)
missing_value_pct_stats = (data.isnull().sum() / len(data)*100)
missing_value_col_count = sum(missing_value_pct_stats > 0)
# missing_value_stats = missing_value_pct_stats.sort_values(ascending=False)[:missing_value_col_count]
missing_data = pd.concat([total, missing_value_pct_stats], axis=1, keys=['Total', 'Percentage(%)'])
print("Number of rows with at least 1 missing values:", data.isna().any(axis = 1).sum())
print("Number of columns with missing values:", missing_value_col_count)
if missing_value_col_count != 0:
# print out column names with missing value percentage
print("\nMissing percentage (desceding):")
display(missing_data[:missing_value_col_count])
# plot missing values
missing = data.isnull().sum()
missing = missing[missing > 0]
missing.sort_values(inplace=True)
missing.plot.bar()
else:
print("No missing data!!!")
# pass a dataframe to the function
missing_value_describe(pulse_point_df)
4 Data Cleaning¶
Discard Columns¶
In [ ]:
pulse_point_df.drop(['id', 'incident_logo', 'agency_logo'], axis=1, inplace=True)
Remove Active Incidents¶
Because active incidents are the noisy duplicated data of the “recent type incident” which was unable to remove during the data collection process. Thus, it does not contribute to the analysis.
In [ ]:
pulse_point_df.type.value_counts()
Out[ ]:
In [ ]:
pulse_point_df.drop(pulse_point_df[pulse_point_df.type == 'active'].index, inplace=True)
pulse_point_df.reset_index(drop=True, inplace=True)
Drop redundant column "type"
In [ ]:
pulse_point_df.drop(columns=['type'], axis=1, inplace=True)
5 Feature Extraction¶
5.1 Location¶
In [ ]:
pulse_point_df.location
Out[ ]:
In [ ]:
pulse_point_df.location.value_counts().head(10)
Out[ ]:
Insights and Feature Extraction¶
There are many variations in the location column:
Such as -
- NE OAK SPRINGS FARM RD, CARLTON, OR
- W 10TH ST, LONG BEACH, CA
- 302 E OJAI AVE, OJAI, CA (OJAI ARCADE (21002302))
- 175 NE 1ST ST, MCMINNVILLE, OR (MCMINNVILLE FIRE DEPARTMENT)
- E BARNETT RD, MEDFORD, OR
We can split the locations into multiple features -
State¶
Text after the last comma appears to be the short form of US states or Canadian provinces.
CA -> California state
OR -> Oregon state
City¶
Text after second last comma appears to be city name (or town, county name)
MEDFORD is a city in Oregon (last example - "E BARNETT RD, MEDFORD, OR")
Address¶
Apart from state and city name, the rest will be counted as address features if there are three comma-separated elements (texts)
Address_2¶
Apart from state, city, and address the rest will be counted as extended address (address_2) feature if there are four comma-separated element/string
Business¶
Bracket enclosed string will be counted as Business Name.
From the above example - OJAI ARCADE (21002302) and MCMINNVILLE FIRE DEPARTMENT are counted as business feature
Business Place Extractor¶
In [ ]:
def get_business_name(location):
# https://stackoverflow.com/a/38212061/11105356
stack = 0
start_index = None
results = []
for i, c in enumerate(location):
if c == '(':
if stack == 0:
start_index = i + 1 # string to extract starts one index later
# push to stack
stack += 1
elif c == ')':
# pop stack
stack -= 1
if stack == 0:
results.append(location[start_index:i])
try:
if len(results) == 0:
return None
elif len(results) == 1 and len(results[0]) == 1:
return None
elif len(results) == 1 and len(results[0])!=1:
return results[0].strip()
elif len(results) > 1 and len(results[0])==1:
return None
else:
return results[1].strip()
except IndexError as ie:
pass
### handles variations such as -
# 5709 RICHMOND RD, STE 76, JAMES CITY COUNTY, VA (JANIE & JACK)
# 433 SARATOGA RD, SCHENECTADY, NY ((GLENVILLE)EAST GLENVILLE FD)
# I 229 RAMP & I 229 RAMP (0.1 MILES), SIOUX FALLS, SD (I 229 MM 8 NB)
# 6501 MISTY WATERS DR, STE (S)E260 (N), BURLEIGH COUNTY, ND
Split Location¶
Example 1 (3 elements):
302 E OJAI AVE, OJAI, CA (OJAI ARCADE (21002302))address =
302 E OJAI AVE, city =OJAI, state =CA, business =OJAI ARCADE (21002302)Example 2 (4 elements):
GRASSIE BLVD, STE 212, WINNIPEG, MANITOBAaddress =
GRASSIE BLVD, address_2 =STE 212, city =WINNIPEG, state =MANITOBA(wil be converted toMBlater)
In [ ]:
# examples
# 302 E OJAI AVE, OJAI, CA (OJAI ARCADE (21002302)) --- 3 segments with business inside
# 1959 MORSE RD, COLUMBUS, OH (DOLLAR GENERAL)
# I 229 RAMP & I 229 RAMP (0.1 MILES), SIOUX FALLS, SD (I 229 MM 8 NB)
# GRASSIE BLVD, STE 212, WINNIPEG, MANITOBA --- 4 segments
# split location into 3 or 4 parts depending on number of commas ->
# 3 segments : address, city, state
# 4 segments : address, address_2, city, state
# to extract bracket enclosed string
pulse_point_df['business'] = pulse_point_df.location.apply(lambda x : get_business_name(x))
### remove enclosed business name from the location string
pulse_point_location_data = pulse_point_df.apply(lambda row : row['location'].replace(str(row['business']), ''), axis=1)
# remove leftover bracket from the business replacemnt
# https://stackoverflow.com/a/49183590/11105356
# remove a (...) substring with a leading whitespace at the end of the string only
pulse_point_location_data = pulse_point_location_data.str.replace(r"\s*\([^()]*\)$","").str.strip()
# split the location
four_col_location_split = ['address', 'address_2', 'city','state']
three_col_location_split = ['address', 'city','state']
# four col indices
# pulse_point_location_data[pulse_point_location_data.str.split(',', expand=True)[3].notna()]
extra_loc_data = pulse_point_location_data.str.split(',', expand=True) # to expand columns
four_col_indices = extra_loc_data[extra_loc_data.apply(lambda x: np.all(pd.notnull(x[3])) , axis = 1)].index
four_col_loc_df = extra_loc_data.iloc[four_col_indices]
four_col_loc_df.columns = four_col_location_split
four_col_loc_df
Out[ ]:
Four Features¶
In [ ]:
pulse_point_df.loc[four_col_loc_df.index , four_col_location_split] = four_col_loc_df
pulse_point_df[four_col_location_split] = pulse_point_df[four_col_location_split].apply(lambda x: x.str.strip())
pulse_point_df[four_col_location_split]
# there are very few numbers of four feature location than three feature location
Out[ ]:
Three Features¶
In [ ]:
four_col_loc_df_mask = extra_loc_data.index.isin(four_col_indices)
three_col_loc_df = extra_loc_data[~four_col_loc_df_mask].drop([3], axis=1)
three_col_loc_df.columns = three_col_location_split
# extra_loc_data[~three_col_loc_df][3].notna().sum() # to check null values
three_col_loc_df
Out[ ]:
In [ ]:
pulse_point_df.loc[three_col_loc_df.index , three_col_location_split] = three_col_loc_df
pulse_point_df[three_col_location_split] = pulse_point_df[three_col_location_split].apply(lambda x: x.str.strip())
pulse_point_df[three_col_location_split]
Out[ ]:
Final Merging of Location Features¶
In [ ]:
pulse_point_df[['location','address', 'address_2', 'city','state', 'business']]
Out[ ]:
In [ ]:
missing_value_describe(pulse_point_df[['location','address', 'address_2', 'city','state', 'business']])
Drop Garbage¶
In [ ]:
pulse_point_df[pulse_point_df.city.isna()]
Out[ ]:
In [ ]:
pulse_point_df = pulse_point_df[pulse_point_df.city.notna()]
5.2 City¶
In [ ]:
mask = ((pulse_point_df.city.isna()) | (pulse_point_df.city==u'') )
display(pulse_point_df[mask])
The business names are same as the city names. I first removed the text containing business names and then performed text extraction for cities. That's why city names are blank for the cases like these.
Let's replace their city names with business names.
In [ ]:
pulse_point_df.loc[mask,'city'] = pulse_point_df[mask].business
5.3 State¶
In [ ]:
display(pulse_point_df.state.value_counts())
printmd(f"**Total {len(pulse_point_df.state.value_counts().index)} States. Some of them are Canadian provinces, ex - MANITOBA**")
In [ ]:
# Canadian Province Mapping
# https://www150.statcan.gc.ca/n1/pub/92-195-x/2011001/geo/prov/tbl/tbl8-eng.htm
# https://en.wikipedia.org/wiki/Provinces_and_territories_of_Canada
ca_province_dic = {
'Newfoundland and Labrador': 'NL',
'Prince Edward Island': 'PE',
'Nova Scotia': 'NS',
'New Brunswick': 'NB',
'Quebec': 'QC',
'Ontario': 'ON',
'Manitoba': 'MB',
'Saskatchewan': 'SK',
'Alberta': 'AB',
'British Columbia': 'BC',
'Yukon': 'YT',
'Northwest Territories': 'NT',
'Nunavut': 'NU',
}
# approach 1
# def handle_state(data_attr):
# for k, v in canada_provinces_dic.items():
# if data_attr.strip().lower() == k.lower():
# return canada_provinces_dic[k]
# else:
# return data_attr
# pulse_point_df['state'] = pulse_point_df.state.apply(handle_state)
# approach 2
# https://stackoverflow.com/a/69994272/11105356
ca_province_dict = {k.lower():v for k,v in ca_province_dic.items()}
pulse_point_df['state'] = pulse_point_df['state'].str.lower().map(ca_province_dict).fillna(pulse_point_df.state)
Noise Removal¶
In [ ]:
# Exception state : example - 'FL #1005' , 'NY EAST GLENVILLE FD', ' DE / RM304'
mask = pulse_point_df.state.apply(lambda x:len(x)>2)
display(pulse_point_df[mask].state)
Keeping only the first segment which is the short form for city, discarding the rest(noise)
In [ ]:
pulse_point_df.loc[mask,'state'] = pulse_point_df[mask].state.apply(lambda x: x.split()[0])
In [ ]:
pulse_point_df.state.value_counts()
Out[ ]:
Leftover¶
In [ ]:
# CONCORD
mask = pulse_point_df.state.str.startswith('CONCORD')
display(pulse_point_df[mask])
printmd("**CONCORD should be in CA**")
In [ ]:
pulse_point_df.loc[mask,'state'] = 'CA'
In [ ]:
pulse_point_df.state.value_counts()
Out[ ]:
In [ ]:
#https://stackoverflow.com/a/57846984/11105356
UNITS = {'s':'seconds', 'm':'minutes', 'h':'hours', 'd':'days', 'w':'weeks'}
# chance of having days and weeks is none
def convert_to_seconds(s):
s = s.replace(" ","")
return int(timedelta(**{
UNITS.get(m.group('unit').lower(), 'seconds'): int(m.group('val'))
for m in re.finditer(r'(?P<val>\d+)(?P<unit>[smhdw]?)', s, flags=re.I)
}).total_seconds())
# convert_to_seconds("1 h 34 m")
In [ ]:
pulse_point_df["duration_in_seconds"] = pulse_point_df.duration.apply(lambda x:convert_to_seconds(x))
Adding Extra Time Related Features¶
In [ ]:
pulse_point_df["day_name"], pulse_point_df["weekday"] = pulse_point_df.date_of_incident.dt.day_name(), pulse_point_df.date_of_incident.dt.weekday
pulse_point_df["month_name"] = pulse_point_df.date_of_incident.dt.month_name()
## more features
# pulse_point_df.date_of_incident.dt.month_name()
# pulse_point_df.date_of_incident.dt.month
# pulse_point_df.date_of_incident.dt.day
# pulse_point_df.date_of_incident.dt.day_name()
# pulse_point_df.date_of_incident.dt.weekday
# pulse_point_df.date_of_incident.dt.isocalendar().week
In [ ]:
pulse_point_df.tail(40)
Out[ ]:
Time of the day¶
I will assign Daytime values based on the time range below -
| Time of the Day | Range |
|---|---|
| Morning | 5 AM to 11:59 AM |
| Afternoon | 12PM to 4:59 PM |
| Evening | 5 PM to 8:59 PM |
| Night | 9 PM to 11:59 PM |
| Midnight | 12 AM to 4:59 AM |
In [ ]:
# https://stackoverflow.com/a/70018607/11105356
def time_range(time):
hour = datetime.strptime(time, '%I:%M %p').hour
if hour > 20:
return "Night"
elif hour > 16:
return "Evening"
elif hour > 11:
return "Afternoon"
elif hour > 4:
return "Morning"
else:
return "Midnight"
In [ ]:
pulse_point_df["time_of_the_day"] = pulse_point_df.timestamp_time.apply(lambda time: time_range(time))
# # pulse_point_df.timestamp_time = pd.to_datetime(pulse_point_df.timestamp_time).dt.time
Save Cleaned and Processed Dataset¶
In [ ]:
pulse_point_df.to_csv('PulsePoint-emergencies-cleaned.csv', index=False)
6 EDA¶
A quick overview of the preprocessed data-
The preprocessed dataset contains additional 5 columns extracted from the location column, another 5 columns extracted from date_of_incident and duration columns. Id , Incident_logo and agency_logo columns from the original dataset was discarded.
| Columns | Description | Data Type |
|---|---|---|
| business | Name of the business place extracted from location(e.g., JANIE & JACK, DOLLAR GENERAL etc.) | object |
| address | Address where the incident took place (extracted from location) | object |
| address_2 | Extended address where the incident took place (extracted from location) | object |
| city | City where the incident took place (extracted from location). It could also be a town or a country | object |
| state | State where the incident took place (extracted from location) | object |
| duration_in_seconds | Incident duration in seconds (extracted from duration) | numeric, int |
| day_name | Name of the day when the incident took place | object |
| weekday | The day of the week with Monday=0, Sunday=6. | object |
| month_name | Name of the month (extracted from date) | object |
| time_of_the_day | morning (5AM-11:59AM), afternoon (12PM-4:59 PM), evening (5PM-8:59PM), night (9PM-11:59PM), midnight (12AM-4:59AM) | object |
In [ ]:
printmd(f"There are total **{pulse_point_df.shape[0]}** incidents")
In [ ]:
pulse_point_df.info()
In [ ]:
pulse_point_df.describe().T
Out[ ]:
In [ ]:
pulse_point_df.describe(include='object').T
Out[ ]:
In [ ]:
missing_value_describe(pulse_point_df)
6.1 Incidents¶
In [ ]:
printmd(f"There are total **{len(pulse_point_df.title.unique())}** types of incidents")
Top¶
In [ ]:
pulse_point_df.title.value_counts().head(20)
Out[ ]:
Wordcloud¶
In [ ]:
# crisp wordcloud : https://stackoverflow.com/a/28795577/11105356
data = pulse_point_df.title.value_counts().to_dict()
wc = WordCloud(width=800, height=400,background_color="white", max_font_size=300).generate_from_frequencies(data)
plt.figure(figsize=(14,10))
plt.imshow(wc, interpolation="bilinear")
plt.axis('off')
plt.show()
6.2 Agency¶
In [ ]:
printmd(f"There are total **{len(pulse_point_df.agency.unique())}** agencies")
Most Active¶
In [ ]:
# Top agencies by incident engagement count
pulse_point_df.agency.value_counts().head(20)
Out[ ]:
In [ ]:
pulse_point_df.agency.value_counts().head(10).sort_values(ascending=False).plot(kind = 'bar');
Wordcloud¶
Most frequent - Montgomery County
In [ ]:
data = pulse_point_df.agency.value_counts().to_dict()
wc = WordCloud(width=800, height=400,background_color="white", max_font_size=300).generate_from_frequencies(data)
plt.figure(figsize=(14,10))
plt.imshow(wc, interpolation="bilinear")
plt.axis('off')
plt.show()
6.3 Description (Codes)¶
The codes themselves are defined by each agency, and are typically followed by a number to identify a particular instance of each asset type. A legend is sometimes provided on the agency information page, and following are some common examples:
- B=Battalion
- BC=Battalion
- Chief E=Engine
- CMD=Command
- CPT=Helicopter
- C=Crew
- DZR=Dozer
- HM=Hazmat
- ME=Medic Engine
- MRE=Medic Rescue Engine
- P=Patrol
- R=Rescue
- RE=Rescue Engine
- SQ=Squad
- T=Truck
- U=Utility
- WT=Water Tender
Credit: PulsePoint Wikipedia
Note: There is no standard for the identifier abbreviations (E, T, S, BC, RA, PM, etc.), and they can vary significantly from agency to agency.
Example - Ventura County Fire Department PulsePoint Unit Abbreviations PDF
To know more, visit - https://www.pulsepoint.org/unit-status-legend
In [ ]:
pulse_point_df.description.value_counts().head(10)
Out[ ]:
Checking for MRE(Medic Rescue Engine) code in description
In [ ]:
mask = pulse_point_df.description.str.contains('MRE', regex=False, na=False)
display(pulse_point_df.description[mask])
printmd(f"**{pulse_point_df.description[mask].count()}** instances contain **MRE** code")
6.4 Duration¶
In [ ]:
(pulse_point_df.duration_in_seconds/ 60).value_counts().head(30)
# alternative
# pulse_point_df.duration.value_counts().head(20)
Out[ ]:
Most of the emergency engagement lasted under 30 mins
6.5 Incident Location¶
City¶
In [ ]:
printmd(f"There are total **{len(pulse_point_df.city.unique())}** cities PulsePoint covered")
In [ ]:
pulse_point_city_df = pulse_point_df.groupby(['city','state'], as_index=False).count()[['city', 'state', 'title']].reset_index(drop=True).rename(columns={'title':'count'})
pulse_point_city_df.head(50)
Out[ ]:
Some cities in different states have the same name
In [ ]:
pulse_point_city_df[pulse_point_city_df.city.str.lower() == 'bloomington']
Out[ ]:
Outliers in city names - * , '0324', ' UNDEFINED', '12809 - NAME?'** etc.
In [ ]:
pulse_point_df[pulse_point_df.city.str.startswith('0324')].head()
Out[ ]:
In [ ]:
geolocator = Nominatim(user_agent='myapplication')
location = geolocator.geocode("50TH ST S")
print(location.address)
display(location.raw)
print("Lattitude: ", location.raw['lat'],", Longitude: ", location.raw['lon'])
Utility Function¶
In [ ]:
geolocator = Nominatim(user_agent='myapplication')
def get_nominatim_geocode(address):
try:
location = geolocator.geocode(address)
return location.raw['lon'], location.raw['lat']
except Exception as e:
# print(e)
return None, None
# alternative way : scraping from the website
# def get_nominatim_geocode(address):
# url = 'https://nominatim.openstreetmap.org/search/' + urllib.parse.quote(address) + '?format=json'
# try:
# response = requests.get(url).json()
# return response[0]["lon"], response[0]["lat"]
# except Exception as e:
# # print(e)
# return None, None
def get_positionstack_geocode(address):
BASE_URL = "http://api.positionstack.com/v1/forward?access_key="
API_KEY = API_KEY_POSITIONSTACK
url = BASE_URL +API_KEY+'&query='+urllib.parse.quote(address)
try:
response = requests.get(url).json()
# print( response["data"][0])
return response["data"][0]["longitude"], response["data"][0]["latitude"]
except Exception as e:
# print(e)
return None, None
def get_geocode(address):
long,lat = get_nominatim_geocode(address)
if long == None:
return get_positionstack_geocode(address)
else:
return long,lat
address = "50TH ST S"
get_geocode(address)
Out[ ]:
Some cities with the same names appear in two different countries.
examples -
- NAPLES - Italy
- Columbia - Country in South America
- Suffolk - UK
- STAFFORD - UK
- NORFOLK - UK
In [ ]:
address = 'Suffolk'
location = geolocator.geocode(address)
location
Out[ ]:
Adding tailing 'USA' to the location text would solve this issue
In [ ]:
address = 'Suffolk, USA'
location = geolocator.geocode(address)
location
Out[ ]:
Adding city and country names will help to get the appropriate location
Let's fetch geolocation of some cities
In [ ]:
test_df = pulse_point_city_df.tail()
test_df
Out[ ]:
In [ ]:
test_df['location'] = test_df['city'] + ', ' + test_df['state'] + ', USA'
# test_df[['city', 'state']].agg(', '.join, axis=1) + ', USA'
test_df
Out[ ]:
In [ ]:
%%time
location_test_df = test_df.location.progress_apply(lambda x:get_geocode(str(x.strip()))).apply(pd.Series)
In [ ]:
location_test_df.columns = ['longitude', 'latitude']
test_df = test_df.join(location_test_df)
display(test_df)
Alternative Approach (iterate every rows)
In [ ]:
%%time
for index,row in test_df.iterrows():
test_df.loc[index,'longitude'], test_df.loc[index,'latitude'] = get_geocode(row.city.strip())
display(test_df)
Create a temporary column "location" by merging city, state and country
In [ ]:
canada_mask = pulse_point_city_df.state.isin([*ca_province_dic.values()])
pulse_point_city_df['location'] = pulse_point_city_df['city'] + ', ' + pulse_point_city_df['state']
pulse_point_city_df['location'].loc[canada_mask] = pulse_point_city_df['location'] + ', CANADA'
pulse_point_city_df['location'].loc[~canada_mask] = pulse_point_city_df['location'] + ', USA'
# to verify
# pulse_point_city_df[pulse_point_city_df['location'].str.endswith('USA')]
# pulse_point_city_df[pulse_point_city_df['location'].str.endswith('CANADA')]
Fetch Geolocation¶
In [ ]:
%%time
location_df = pulse_point_city_df.location.progress_apply(lambda x:get_geocode(str(x.strip()))).apply(pd.Series)
In [ ]:
location_df.columns = ['longitude', 'latitude']
pulse_point_city_df = pulse_point_city_df.join(location_df)
Check for missing values and drop cities without geocode
In [ ]:
pulse_point_city_df.isna().sum()
Out[ ]:
In [ ]:
pulse_point_city_df.dropna(inplace=True)
In [ ]:
pulse_point_city_df.to_csv('City-coordinate.csv', index=False)
In [ ]:
pulse_point_city_df.tail()
Out[ ]:
Top Engagement¶
In [ ]:
pulse_point_city_df.sort_values(by='count', ascending=False).head(20)
Out[ ]:
Top 5 Cities by agency engagement -
| Name | Count | State |
|---|---|---|
| 1. LOS ANGELES | 7449 | CA |
| 2. MILWAUKEE | 4404 | WI |
| 3. COLUMBUS | 4115 | OH |
| 4. CLEVELAND | 3977 | OH |
| 5. ROCKFORD | 2950 | IL |
Heat Map¶
In [ ]:
geometry = geopandas.points_from_xy(pulse_point_city_df.longitude, pulse_point_city_df.latitude)
geo_df = geopandas.GeoDataFrame(pulse_point_city_df[['city','count','longitude', 'latitude']], geometry=geometry)
geo_df.head()
Out[ ]:
In [ ]:
map = folium.Map(location = [48, -102], tiles='Cartodb dark_matter', zoom_start = 4)
heat_data = [[point.xy[1][0], point.xy[0][0]] for point in geo_df.geometry ]
# heat_data
HeatMap(heat_data).add_to(map)
map
Out[ ]:
Map Circle Overlays¶
In [ ]:
# to avoid recursion depth issue change latitude,longitude type to float
# https://github.com/python-visualization/folium/issues/1105
pulse_point_city_df['latitude'] = pulse_point_city_df['latitude'].astype(float)
pulse_point_city_df['longitude'] = pulse_point_city_df['longitude'].astype(float)
In [ ]:
map_USA = folium.Map(location=[48, -102],
zoom_start=4,
prefer_canvas=True,
)
occurences = folium.map.FeatureGroup()
n_mean = pulse_point_city_df['count'].mean()
for lat, lng, number, city in zip(pulse_point_city_df['latitude'],
pulse_point_city_df['longitude'],
pulse_point_city_df['count'],
pulse_point_city_df['city']):
occurences.add_child(
folium.vector_layers.CircleMarker(
[lat, lng],
radius=number/(n_mean/3), # radius for number of occurrences
color='yellow',
fill=True,
fill_color='blue',
fill_opacity=0.4,
# tooltip = city
tooltip=str(number)+','+str(city)[:21], # can be displayed max 21 character
# most of the city names contain 5-20 characters
# check pulse_point_city_df.city.apply(len).plot();
# get more from tooltip https://github.com/python-visualization/folium/issues/1010#issuecomment-435968337
)
)
map_USA.add_child(occurences)
Out[ ]:
State¶
In [ ]:
printmd(f"There are total **{len(pulse_point_df.state.unique())}** US states & Canadian provinces PulsePoint covered")
Top Engagement¶
In [ ]:
pulse_point_df.state.value_counts().head(20)
Out[ ]:
Top 5 States by agency engagement -
| Name | Count | Abbreviation |
|---|---|---|
| 1. California | 70989 | CA |
| 2. Florida | 23213 | FL |
| 3. Virginia | 16016 | VA |
| 4. Washington | 15532 | WA |
| 5. Ohio | 14440 | OH |
Let's Visualize it
In [ ]:
pulse_point_df.state.value_counts().head(10).sort_values(ascending=False).plot(kind = 'bar');
Duration¶
In [ ]:
(pulse_point_df.groupby('state').sum()['duration_in_seconds'].sort_values(ascending=False) / 3600).head(5)
Out[ ]:
California has over 43,000 hours of agency engagement whereas the second highest Florida has over 15,000 hours of engagement which is 3 times less
Occurrence Timeline¶
In [ ]:
df_state_incident = pulse_point_df.groupby(["date_of_incident",
"state"],
as_index=False).count()[['date_of_incident',
'state', 'title']].reset_index(drop=True).rename(columns={'date_of_incident':'date',
'title':'count'})
df_state_incident.columns = ['date', 'state', 'count']
df_state_incident
Out[ ]:
In [ ]:
pipeline = pdp.PdPipeline([
pdp.ApplyByCols('count', set_size, 'size', drop=False),
])
agg_incident_data = pipeline.apply(df_state_incident)
agg_incident_data.fillna(0, inplace=True)
agg_incident_data = agg_incident_data.sort_values(by='date', ascending=True)
agg_incident_data.date = agg_incident_data.date.dt.strftime('%Y-%m-%d') # convert to string object
agg_incident_data.tail()
Out[ ]:
Animated geo scatter plot¶
In [ ]:
fig = px.scatter_geo(
agg_incident_data, locations="state", locationmode='USA-states',
scope="usa",
color="count",
size='size', hover_name="state",
range_color= [0, 2000],
projection="albers usa", animation_frame="date",
title='PulsePoint Incidents: Local Emergencies By State',
color_continuous_scale="portland"
)
fig.show()
In [ ]:
# https://developers.google.com/public-data/docs/canonical/states_csv
state_coordinate = pd.read_html("https://developers.google.com/public-data/docs/canonical/states_csv")[0]
state_coordinate
Out[ ]:
US States with Total Incident Count¶
In [ ]:
pulse_point_state_df = pulse_point_df.groupby(['state']).count()[['title']].reset_index().rename(columns={'title':'count'})
pulse_point_state_df
Out[ ]:
Missing US States¶
In [ ]:
state_coordinate[~state_coordinate.state.isin(pulse_point_state_df.state)].reset_index(drop=True)
Out[ ]:
Filter US States¶
In [ ]:
pulse_point_state_df = pulse_point_state_df.merge(state_coordinate, on='state', how='left')
pulse_point_state_df
# there are three provinces of canada :
# Manitoba : MB
# British Columbia : BC
# Ontario : ON
Out[ ]:
Drop Canadian Provinces
In [ ]:
pulse_point_state_df.dropna(inplace=True)
pulse_point_state_df = pulse_point_state_df.reset_index(drop=True)
Choropleth USA¶
In [ ]:
url = (
"https://raw.githubusercontent.com/python-visualization/folium/master/examples/data"
)
state_geo = f"{url}/us-states.json"
state_data = pulse_point_state_df.iloc[:,[0,1]]
m = folium.Map(location=[48, -102], zoom_start=4)
folium.Choropleth(
geo_data=state_geo,
name="choropleth",
data=state_data,
columns=["state", "count"],
key_on="feature.id",
fill_color="YlGn",
fill_opacity=0.7,
line_opacity=0.2,
legend_name="Number of Incidents",
).add_to(m)
folium.LayerControl().add_to(m)
m
Out[ ]:
With Count Marker¶
In [ ]:
# icon credit : https://icon-icons.com/icon/location-sos-phone-call-help/68848
# https://www.clipartmax.com/middle/m2H7i8G6N4H7b1N4_metallic-icon-royalty-free-cliparts-icone-sos-png/
# custom icon : https://stackoverflow.com/a/68992396/11105356
for i in range(0, len(pulse_point_state_df)):
folium.Marker(
location = [pulse_point_state_df.iloc[i]['latitude'], pulse_point_state_df.iloc[i]['longitude']],
popup = folium.Popup(f"{pulse_point_state_df.iloc[i]['name']}\n{pulse_point_state_df.iloc[i]['count']}", parse_html=True),
icon=folium.features.CustomIcon('https://i.postimg.cc/JhmnMQXj/sos.png', icon_size=(24, 31))
).add_to(m)
m
Out[ ]:
Choropleth USA Zoomed-IN¶
In [ ]:
# https://plotly.com/python/choropleth-maps
fig = go.Figure(data=go.Choropleth(
locations=pulse_point_state_df['state'], # Spatial coordinates
z = pulse_point_state_df['count'].astype(float), # Data to be color-coded
locationmode = 'USA-states', # set of locations match entries in `locations`
colorscale = 'Reds',
colorbar_title = "Total Occurrences",
))
fig.update_layout(
title_text = 'US PulsePoint Emergencies Occurrences by State',
geo_scope='usa', # limite map scope to USA
)
fig.show()
6.6 Incident Date & Time¶
In [ ]:
pulse_point_df.date_of_incident.value_counts().head(20)
Out[ ]:
Timeline¶
Daily¶
In [ ]:
pulse_point_df.groupby(['date_of_incident']).count()['title'].reset_index().rename(columns={'date_of_incident':'Date',
'title':'count'}).sort_values('Date').plot(y='count',
x='Date',label="Incident")
plt.xlabel('Date of Incidents')
plt.ylabel('Number of Incidents')
plt.title("Incidents Frequency (Daily)")
plt.show();
Number of PulsePoint dispatches increased after August, 2021
Weekly¶
In [ ]:
pulse_point_df.groupby([pd.Grouper(key='date_of_incident',
freq='W-MON')]).count()['title'].reset_index().rename(columns={'date_of_incident':'Date',
'title':'count'}).sort_values('Date').plot(y='count',
x='Date',label="Incident")
plt.xlabel('Date (Month)')
plt.ylabel('Number of Incidents')
plt.title("Incidents Frequency (Weekly)")
plt.show();
Weekday Time¶
In [ ]:
# pulse_point_df.groupby(['day_name','time_of_the_day'],as_index=False).count()
incident_time_df = pulse_point_df.groupby(["day_name", "time_of_the_day"],
as_index=False).count()[['day_name',
'time_of_the_day',
'title']].reset_index(drop=True).rename(columns={'date_of_incident':'date',
'title':'incident_count'})
incident_time_df
Out[ ]:
In [ ]:
display(incident_time_df.groupby(['day_name']).sum().reset_index().sort_values(["incident_count"], ascending=False))
display(incident_time_df.groupby(['day_name']).sum().plot(kind='bar'))
Highest number of incidents occurred on Sunday
Time of the Day¶
In [ ]:
# incident_time_df.groupby(['day_name','time_of_the_day']).sum().plot(kind='bar', figsize=(25,8));
fig = px.bar(incident_time_df,
x="day_name",
y="incident_count",
color="time_of_the_day",
barmode="group",
labels={'day_name':'Day',
'incident_count': 'Incident Count',
'time_of_the_day': ''},
title=f"Number of Incidents by Time of the Day",
).for_each_trace(lambda t: t.update(name=t.name.replace("=","")))
printmd("Emergency responses spiked at **midnight** or in the **morning**")
fig.show()
## alternative in seaborn catplot
# g=sns.catplot(data= incident_time_df,
# x="time_of_the_day",
# col='day_name',
# y='incident_count',
# kind='bar',
# height=6,
# col_wrap=4,
# )
# # bug : x-ticks not showing while using col_wrap
# # fixed : https://stackoverflow.com/a/52184614/11105356
# for ax in g.axes.flatten():
# ax.tick_params(labelbottom=True)
# g.set_ylabels('Incident count')
# # for rotated x-ticks
# # for ax in g.axes:
# # plt.setp(ax.get_xticklabels(), visible=True, rotation=45)
# # plt.subplots_adjust(hspace=0.5)
# plt.show()
Most of the incidents occurred during Midnight or in the morning. Probably some of the incidents were already started at nighttime and were logged later in the morning.
6.7 Major Incidents¶
In [ ]:
pulse_point_df.groupby(['title']).count()[['agency']].rename(columns={'agency':'total'}).sort_values('total', ascending=False)[:25]
Out[ ]:
In [ ]:
mask = (pulse_point_df.time_of_the_day == 'Midnight') | (pulse_point_df.time_of_the_day == 'Morning')
highest_occ_incident = pulse_point_df[mask].groupby(['time_of_the_day','title']).count()[['agency']].rename(columns={'agency':'total'})
highest_occ_incident.sort_values('total', ascending=False)[:25]
Out[ ]:
Top ten emergencies during 'Midnight' or 'Morning' -
Midnight :
- Medical Emergency
- Traffic Collision
- Fire Alarm
- Alarm
- Public Service
- Structure Fire
- Refuse/Garbage Fire
- Mutual Aid
- Residential Fire
- Expanded Traffic Collision
Morning :
- Medical Emergency
- Traffic Collision
- Fire Alarm
- Public Service
- Refuse/Garbage Fire
- Structure Fire
- Fire
- Residential Fire
- Mutual Aid
- Lift Assist
7 Clustering¶
7.1 Preprocess Dataset¶
In [ ]:
pulse_point_df.isna().sum()
Out[ ]:
Filter Data¶
- "timestamp_time" was replaced with "time_of_the_day" feature
- "date_of_incident" was replaced with "week_day", "day_name" and "month_name"
- "business" and "address_2" has lots of null values, hence those features were removed
- "duration" was converted to numerical value and replaced with "duration_in_seconds"
In [ ]:
pulse_point_cluster_df = pulse_point_df.drop([# 'location',
'timestamp_time',
'date_of_incident',
# 'description',
'duration',
# 'address',
'business',
'address_2',
], axis=1)
pulse_point_cluster_df.dropna(inplace=True)
pulse_point_cluster_df.isna().sum()
Out[ ]:
Scaling¶
In [ ]:
def scaling_df(df):
X_cluster = df.copy()
object_cols = df.columns[df.dtypes == object].to_list()
label_enc=LabelEncoder()
for i in object_cols:
X_cluster[i]=X_cluster[[i]].apply(label_enc.fit_transform)
scaler = MinMaxScaler()
scaler.fit(X_cluster)
X_cluster_scaled = pd.DataFrame(scaler.transform(X_cluster),columns= X_cluster.columns)
return X_cluster_scaled
In [ ]:
X_cluster = pulse_point_cluster_df.copy()
X_cluster_scaled = scaling_df(X_cluster)
X_cluster_scaled
Out[ ]:
PCA¶
In [ ]:
def pulse_point_pca(X_data, n_components):
pca = PCA(n_components=n_components)
fit_pca = pca.fit(X_data)
print("Variance Explained with {0} components ".format(n_components),
round(sum(fit_pca.explained_variance_ratio_),2))
return fit_pca, fit_pca.transform(X_data)
In [ ]:
# for 12 components
pca_full, pulsepoint_data_full = pulse_point_pca(X_cluster_scaled, X_cluster_scaled.shape[1])
In [ ]:
X_cluster_scaled.shape
Out[ ]:
In [ ]:
plt.plot(np.cumsum(pca_full.explained_variance_ratio_))
plt.title("Proportion of PCA variance\nexplained by number of components")
plt.xlabel("Number of components")
plt.ylabel("Proportion of variance explained");
We need about 7 components to explain ~90% of the variance in the data
7.2 Agency Engagement Vs Incident Duration by City¶
In [ ]:
pulse_point_state_duration_df = pulse_point_cluster_df.groupby('city').agg({'agency':'count', 'duration_in_seconds': 'sum'}).reset_index()
pulse_point_state_duration_df.duration_in_seconds = pulse_point_state_duration_df.duration_in_seconds.apply(lambda x: x/3600)
pulse_point_state_duration_df.columns= ['city','total_agency_engagement', 'total_duration_hr']
In [ ]:
x = pulse_point_state_duration_df['total_agency_engagement'].values
y = pulse_point_state_duration_df['total_duration_hr'].values
plt.scatter(x,y)
plt.title('Agency Engagement vs Incident Duration')
plt.xlabel('Number of Agency Engagement')
plt.ylabel('Total Incident Duration(hour)')
plt.show()
The agency_count (number of agency engagement) and duration_hr (duration in hour) has a positive linear relationship.
Higher Duration of Incidents indicates more agency engagement in a city
Clustering States By Duration¶
In [ ]:
X = pulse_point_state_duration_df[['total_agency_engagement', 'total_duration_hr']].values
K-means Clustering¶
With “kmeans++” initialization the objective of this clustering is to create groups based on number of agency engagements and total incident duration (hour)
In [ ]:
# To decide the optimum cluster number, KMeans++ using Elbow method
# to figure out K for KMeans, I will use ELBOW Method on KMEANS++ Calculation
wcss=[]
for i in range(1,11):
kmeans = KMeans(n_clusters= i, init='k-means++', random_state=SEED)
kmeans.fit(X)
wcss.append(kmeans.inertia_)
# inertia_ is the formula used to segregate the data points into clusters
The best number of K lies between 2 and 4
In [ ]:
plt.plot(range(1,11), wcss)
plt.title('The Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('wcss')
plt.show()
In [ ]:
kmeans = KMeans(n_clusters= 3, init='k-means++', random_state=SEED)
y_kmeans= kmeans.fit_predict(X)
In [ ]:
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='viridis')
centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='blue', s=200, alpha=0.5);
plt.title('Clusters of Incidents By Duration and Agency Engagement')
plt.xlabel('Number of Agency Engagement')
plt.ylabel('Total Incident Duration(hour)')
plt.show()
The k-mean clustering algorithm clusters the cities based on duration of incidents and number of agencies into three groups. Small duration indicates having less number of agency engagement and vice-versa.
Group 1 : Cities with very low number of incidents duraion and agency engagements
Group 2 : Cities with comparatively higher number of incidents duraion and agency engagements
Group 2 : Cities with highest number of incidents duraion and agency engagements
Agglomerative Clustering¶
With “KElbowVisualizer” from yellowbrick library it’s found that optimum K value is 4 ( it implements the “elbow” method)
In [ ]:
# Instantiate the clustering model and visualizer
model = AgglomerativeClustering()
visualizer = KElbowVisualizer(model, k=(1,12))
visualizer.fit(X) # Fit the data to the visualizer
visualizer.show();
Ward Linkage¶
In [ ]:
#Initiating the Agglomerative Clustering model
AC = AgglomerativeClustering(n_clusters=4)
yhat_AC = AC.fit_predict(X)
plt.scatter(X[:, 0], X[:, 1], c=yhat_AC, s=50, cmap='viridis')
plt.title('Clusters of Incidents By Duration and Agency Engagement')
plt.xlabel('Number of Agency Engagement')
plt.ylabel('Total Incident Duration(hour)')
plt.show()
Complete Linkage¶
In [ ]:
#Initiating the Agglomerative Clustering model
AC = AgglomerativeClustering(n_clusters=4, linkage='complete')
yhat_AC = AC.fit_predict(X)
plt.scatter(X[:, 0], X[:, 1], c=yhat_AC, s=50, cmap='viridis')
plt.title('Clusters of Incidents By Duration and Agency Engagement')
plt.xlabel('Number of Agency Engagement')
plt.ylabel('Total Incident Duration(hour)')
plt.show()
Results¶
From the above clustering techniques, it is clear that “complete” linkage is not suitable for Agglomerative clustering (cluster parameter was given 4 but it mostly formed 2 clusters). On the other hand, k-means and “Ward” Agglomerative provided a better clustering result. But the density of the cities is high when the value of number of agency engagement and total incident duration is low.
K-means++ focused on clustering lower dense cities with unequal parameter distribution –
- Cluster 0: Total incident duration = ~400, number of engagements = ~500
- Cluster 1: Total incident duration = 400 to ~1200, number of engagements = 500 to ~1500
- Cluster 2: Total incident duration = 1200 to max, number of engagements = 1500 to max
The range of cluster 1 is bigger than cluster 0 in k-means whereas Ward Agglomerative did almost an equally distributed clustering for cluster 0 and 1. If the range of the parameters (engagements of duration) is important based on other factor, for example – budget allocation with respect to engagements or business decision/future planning based on duration of emergencies, then depending on the priority both clusters would be acceptable.
8 Key Insights¶
- Most of the incidents occurred in California
- Most incidents happened during midnight and in the morning throughout the week
- Most of the emergency engagement lasted under 30 mins
- The highest number of incidents happened on Sunday
- The incidents’ number got increased after Covid-19 lockdown
- Medical emergency was the highest occurring incident which was followed by traffic collision and fire alarm
- Montgomery County, Milwaukee Fire, and Columbus Fire were the top active agencies during the five monthly period
9 Top 10 Incidents¶
In [ ]:
top_incidents_list = pulse_point_df.title.value_counts().head(25).reset_index().rename(columns={'title':'total','index':'title'})
top_incidents_list
Out[ ]:
In [ ]:
top_incidents_list[:10].plot(x='title', y='total', rot=30);
In [ ]:
top_10_incidents = top_incidents_list.title.tolist()[:10]
top_10_incidents
Out[ ]:
In [ ]:
pulse_point_top_10_df = pulse_point_df[pulse_point_df['title'].isin(top_10_incidents)].reset_index(drop=True)
pulse_point_top_10_df
Out[ ]:
Group By Day & Time¶
In [ ]:
top_10_time_df = pulse_point_top_10_df.groupby(["title","day_name", "time_of_the_day"],
as_index=False).count()[['title','day_name',
'time_of_the_day',
'agency']].reset_index(drop=True).rename(columns={'date_of_incident':'date',
'agency':'incident_count'})
top_10_time_df
Out[ ]:
Utility Plot Function¶
In [ ]:
# plotly categorical barplot
def plot_top_incident_by_time(title):
fig = px.bar(top_10_time_df[top_10_time_df.title.str.strip()==title],
x="day_name",
y="incident_count",
color="time_of_the_day",
barmode="group",
labels={'day_name':'Day',
'incident_count': 'Incident Count',
'time_of_the_day': ''},
title=f"{title} by Time of The Day",
).for_each_trace(lambda t: t.update(name=t.name.replace("=","")))
# remove '=' sign from color
# https://github.com/plotly/plotly_express/issues/36
fig.show()
# seaborn alternative
# def plot_top_incident_by_time(title):
# g=sns.catplot(data=top_10_time_df[top_10_time_df.title.str.strip()==title],
# x="time_of_the_day",
# y='incident_count',
# col='day_name',
# kind='bar',
# height=6,
# col_wrap=4,)
# for ax in g.axes.flatten():
# ax.tick_params(labelbottom=True)
# g.set_ylabels('Incident count')
# g.set_axis_labels('Time of the Day')
# g.set_titles("{col_name}")
# # g.despine(left=True)
# # plt.suptitle('Incident By Time')
# plt.show()
Medical Emergency¶
In [ ]:
plot_top_incident_by_time('Medical Emergency')
Traffic Collision¶
In [ ]:
plot_top_incident_by_time('Traffic Collision')
Fire Alarm¶
In [ ]:
plot_top_incident_by_time('Fire Alarm')
Alarm¶
In [ ]:
plot_top_incident_by_time('Alarm')
Public Service¶
In [ ]:
plot_top_incident_by_time('Public Service')
Refuse/Garbage Fire¶
In [ ]:
plot_top_incident_by_time('Refuse/Garbage Fire')
Structure Fire¶
In [ ]:
plot_top_incident_by_time('Structure Fire')
Mutual Aid¶
In [ ]:
plot_top_incident_by_time('Mutual Aid')
Lift Assist¶
In [ ]:
plot_top_incident_by_time('Lift Assist')
Fire¶
In [ ]:
plot_top_incident_by_time('Fire')
10 Top State : CA¶
According to a 2017 study from the U.S. Census Bureau, California state's local governments consist of 57 counties, 482 cities, towns, and villages, and 2,894 special districts
In [ ]:
pulse_point_ca_df = pulse_point_df[pulse_point_df.state.str.strip() == 'CA'].copy()
pulse_point_ca_df.drop(axis=1, columns=['state'],inplace=True)
pulse_point_ca_df.head()
Out[ ]:
10.1 Descriptive Analysis¶
In [ ]:
pulse_point_ca_df.describe().T
Out[ ]:
The average duration of the incidents is ~37 minutes
In [ ]:
pulse_point_ca_df.describe(include='object').T
Out[ ]:
CA State Summary¶
- Contra Costa FPD is the most engaged agency
- Highest reported incidents occurred on Sunday
- After the end of covid-19 lockdown, the number of emergency incidents was higher and peaked at last month - November
- Most reported emergencies have taken place in Los Angeles
- Fire Truck Engine 51 (E51) was the most frequent emergency code
In [ ]:
pulse_point_ca_df.description.value_counts().head(20)
Out[ ]:
Top emergency code descriptions refer to Fire Emergency -
- E51 Fire Engine
- E57 responding to a emergency
- Here is an E53 Fire Truck -
In [ ]:
printmd(f"There are total **{len(pulse_point_ca_df.city.unique())}** cities in California")
In [ ]:
pulse_point_ca_df.day_name.value_counts().head(10)
printmd(f"**Most emergencies take place in Saturday and Sunday (Holiday) in California**")
Incidents by Time of The Day¶
In [ ]:
pulse_point_ca_df.time_of_the_day.value_counts()
Out[ ]:
Major Incidents¶
In [ ]:
pulse_point_ca_df.title.value_counts().head(20)
Out[ ]:
Apart from medical emergency, top incidents of California includes fire, alarm and traffic collision
California is susceptible to an impressive array of natural hazards, including earthquakes, fires, flooding and mudslides.
Here is a good article on this -
4 REASONS CALIFORNIA IS MORE SUSCEPTIBLE TO NATURAL DISASTERS THAN OTHER STATES
Major Cities¶
In [ ]:
pulse_point_ca_df.city.value_counts().head(20)
Out[ ]:
Map Geolocation
In [ ]:
mask = (pulse_point_city_df.city.isin(pulse_point_ca_df.city.unique().tolist())) & (pulse_point_city_df.state == 'CA')
ca_city = pulse_point_city_df[mask].reset_index(drop=True)
ca_city
Out[ ]:
Incidents Distribution¶
In [ ]:
ca_city['count'].describe()
Out[ ]:
In [ ]:
ca_city['count'].plot(title='Emergency Incidents Distribution on California Cities');
plt.xlabel('Number of Cities')
plt.ylabel('Incidents')
plt.show()
10.2 Geoplot¶
Color Code Cities (By Number of Inidents)¶
In [ ]:
ca_city['color']=ca_city['count'].apply(lambda count:"Black" if count>=1500 else
"green" if count>=1200 and count<1500 else
"Orange" if count>=800 and count<1200 else
"darkblue" if count>=500 and count<800 else
"red" if count>=300 and count<500 else
"lightblue" if count>=100 and count<300 else
"brown" if count>=10 and count<100 else
"violet" if count>=5 and count<10 else
"grey")
ca_city['size']=ca_city['count'].apply(lambda count:10 if count>=1500 else
8 if count>=1200 and count<1500 else
7 if count>=800 and count<1200 else
6 if count>=500 and count<800 else
5 if count>=300 and count<500 else
4 if count>=100 and count<300 else
3 if count>=10 and count<100 else
2 if count>=5 and count<10 else
1)
In [ ]:
geometry2 = geopandas.points_from_xy(ca_city.longitude, ca_city.latitude)
geo_df2 = geopandas.GeoDataFrame(ca_city[['city','count','longitude', 'latitude']], geometry=geometry2)
geo_df2.head()
Out[ ]:
In [ ]:
geoJSON_df = geopandas.read_file(state_geo)
geoJSON_CA = geoJSON_df.loc[geoJSON_df.id == 'CA']
geoJSON_CA
Out[ ]:
In [ ]:
map_CA = folium.Map(location = [38, -115], zoom_start = 6)
# https://stackoverflow.com/a/61129097/11105356
folium.GeoJson(geoJSON_CA.geometry,
name='California').add_to(map_CA)
for lat,lon,area,color,count,size in zip(ca_city['latitude'],ca_city['longitude'],ca_city['city'],ca_city['color'],ca_city['count'],ca_city['size']):
folium.CircleMarker([lat, lon],
popup=folium.Popup(f"{area}, {count}", parse_html=True),
radius=size*5,
color='b',
fill=True,
fill_opacity=0.7,
fill_color=color,
).add_to(map_CA)
map_CA
Out[ ]:
With Heatmap¶
In [ ]:
heat_data = [[point.xy[1][0], point.xy[0][0]] for point in geo_df2.geometry ]
# # heat_data
HeatMap(heat_data).add_to(map_CA)
map_CA
Out[ ]:
With Marker¶
In [ ]:
map_CA_c = folium.Map(location = [38, -115], zoom_start = 6)
# https://stackoverflow.com/a/61129097/11105356
folium.GeoJson(geoJSON_CA.geometry,
name='California').add_to(map_CA_c)
for i in range(0, len(ca_city)):
folium.Marker(
location = [ca_city.iloc[i]['latitude'], ca_city.iloc[i]['longitude']],
popup = folium.Popup(f"{ca_city.iloc[i]['city']}\n{ca_city.iloc[i]['count']}", parse_html=True),
icon=folium.features.CustomIcon('https://i.postimg.cc/JhmnMQXj/sos.png', icon_size=(24, 31))
).add_to(map_CA_c)
# # heat_data
HeatMap(heat_data).add_to(map_CA_c)
map_CA_c
Out[ ]:
10.3 Insights¶
- Southern and central California are two most active region
- Most of the incidents have taken place in Los Angelos and San Francisco, both are highly active city (PulsePoint is based in the San Francisco Bay Area)
11 Conclusions¶
Future Work¶
- Dataset still has some outliers in location related features. This issue will be resolved in the future version
- I performed clustering on incident duration and agency engagement grouped by cities. So, different groups of data could be explored for clustering as well as other clustering techniques
- Better result could be achieved by performing dimensionality reduction for clustering or hyperparameter tuning
- Detailed time series & geospatial analysis on the dataset
- The impact of/correlations with the lockdown and other Covid-19-driven policy actions/agency activities in different states in 2020
- Detailed investigation on the possible correlation/inference with the incidents and local infrastructure
End Notes¶
- Demographic data would be valuabe to do research on users to find more insights and their association with emergencies
- A glossary / clear explanation of all emegercy codes (description) would be valuable to sort out incidents
- Research work on PulsePoint can be found here - https://www.pulsepoint.org/research-studies